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A SIGNAL PROCESSING APPROACH FOR CHANNEL CODING 
BASED ON INTER-SYMBOL-INTERFERENCE INSERTION 

5 Background of the Invention 

1. Field of the Invention 

This invention relates generally to digital communication systems and methods, 
and more particularly to a technique of channel encoding that deliberately inserts inter- 
symbol-interference (ISI) into the signal using a non-linear precoder to map data into a 
10 lattice that can have much better distance spectrum than an uncoded QAM signal, 
although it has nearly the same power level. 

2. Description of the Prior Art 

The channel coding problem is one of the key problems of the area of digital 
15 communications. There are numerous existing algebraic coding approaches directed to 
the channel coding problem, including block coding, lattice coding, trellis coding, 
convolutional coding and turbo coding, offering a tradeoff between robustness to 
channel's noise to power level, complexity and delay. 
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Summary of the Invention 

The present invention is directed to a technique of channel encoding that 
deliberately inserts inter-symbol-interference (ISI) into the signal using a non-linear 
precoder to map data into a lattice that can have a much better distance spectrum than an 
uncoded QAM signal, although it has nearly the same power level. 

According to one embodiment, a inter-symbol-interference (ISI) coder comprises: 
a mapper operational to map at least one input symbol sequence into a set of 

multiple sequences of extended symbols; and 

a linear ISI fiher operational to generate coded output symbols in response to the 

set of multiple sequences of extended symbols. 

According to another embodiment, a method of channel coding comprises the 
steps of: 

providing a non-linear precoder; and 

inserting deliberate inter-symbol-interference (ISI) into an input signal via the 
non-linear precoder such that input data is mapped into a lattice having a better distance 
spectrum than an uncoded QAM signal while retaining a substantially unchanged power 
level from the uncoded QAM signal. 
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Brief Description of the Drawings 

Other aspects and features of the present invention and many of the attendant 
advantages of the present invention will be readily appreciated as the invention becomes 
5 better understood by reference to the following detailed description when considered in 
connection with the accompanying drawing figures thereof and wherein: 

Figure la illustrates a simple ISI coder; 

10 Figure lb illustrates an equivalent implementation of the simple ISI coder shown 

in Figure la; 



Figure 2 illustrates an efficient implementation of the simple ISI coder shown in 
Figures la and lb; 

Figure 3 illustrates a generalized ISI coder; 

Figure 4 is a graphic interpretation of the generalized ISI coder shown in Figure 3; 



15 



20 Figure 5 illustrates simulation test results associated with the ISI coder shown in 

Figure 1 showing Event Error Rate versus the Signal to Noise Ratio for an ISI pattern 

F{z) = 1 + 0.8z"' and an 8-PAM input; 



Figure 6 illustrates simulation test results associated with the ISI coder shown in 
25 Figure 1 showing Event Error Rate versus the Signal to Noise Ratio for an ISI pattern 

F{z) that has two zeros at z = -0.9e^''^^ , and an 64-QAM input; 

Figure 7 illustrates simulation test results associated with the ISI coder shown in 
Figure 1 showing Event Error Rate versus the Signal to Noise Ratio of a T-algorithm 

30 RSSD for an ISI pattern F{z) that has three zeros at z = -0.95^^''''^ ; and 
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Figure 8 illustrates simulation test results associated with the ISI coder shown in 
Figure 1 showing Event Error Rate versus the Signal to Noise Ratio of a sequential T- 
algorithm RSSD for an ISI pattern F(z) that has three zeros at z = -0,95e^'''^^ . 

While the above-identified drawing figures set forth particular embodiments, 
other embodiments of the present invention are also contemplated, as noted in the 
discussion. In all cases, this disclosure presents illustrated embodiments of the present 
invention by way of representation and not limitation. Numerous other modifications and 
embodiments can be devised by those skilled in the art which fall within the scope and 
spirit of the principles of this invention. 
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Detailed Description of the Preferred Embodiments 



The preferred embodiments described herein below are directed to a new 
approach for channel coding, called Liter-Symbol-Interference (ISI) coding. ISI coding 

5 is achieved by inserting deliberate Inter-Symbol-Interference into the signal using a non- 
linear precoder. This maps the data into a lattice that can have a much better distance 
spectrum than an uncoded QAM signal, although it has nearly the same power level. The 
ISI encoder may also pre-equalize the channel's linear distortion, as well as shape the 
signal, allowing up to 1 .53 dB shaping gain. Error spectrum analysis has shown that 

10 many simple ISI pattems achieve coding gains that allow reliable operation at data rates 
that are very near Shannon's bound. The optimal decoder for this code was found by the 
present inventor to be Forney's MLSD (Maximum Likelihood Sequence Detector). 
Reduced complexity MLSD approximations, such as the list algorithm and the Fano 
algorithm, were found to achieve the cutoff rate bound (1.68 dB from Shannon Capacity) 

15 with a moderate computational effort. 



ISI codes employ signal-processing algorithms from the area of equalization to 
construct a lattice of signals that have large distances between them, and have nearly the 
same power as uncoded PAM or QAM signals. This lattice is constructed by inserting 

20 Inter-Symbol-Interference (ISI) into the signal using a non-linear precoder, as stated 
herein before. The present inventor found that coding gains of ISI codes in an Additive 
Gaussian Noise (AGN) channel can exceed 7 dB, allowing reliable operation only a 
fraction of a dB from Shannon's bound. These gains may be achieved by Fomey's 
Maximum Likelihood Sequence Detector (MLSD) receiver [Fomey, MLSE], as stated 

25 herein before. The present inventor found that noise immunity could be improved by 5.5- 
6 dB, to reach the cutoff rate of the channel by using practical reduced complexity 
approximations of the MLSD. 



An ISI coder is first described herein below, followed by a description of a Quasi- 
30 MLSD decoder, and its reduced complexity approximations. The error probability of 
these decoders is presented, followed by a discussion of the coding gain attributed to 
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15 



some ISI codes. A discussion of simulation results are finally set forth that verify the 
analytical results. 



The ISI-Coder 

5 Consider first the effect of linear filtering on the minimum distance of a symbol 

sequence. Let 

1^0 



10 be the output of a monic causal filter F(z) whose input is a sequence of i.i.d. (independent 
and identically distributed) uncoded data symbols drawn fi-om an M-PAM constellation 

{'(M-l),„,,-3, -1, 1, 3,„.,M'J}' Denote by and c?/ the minimum distances of the 
sequences a(n) and x(n), respectively. Obviously, is simply the minimum distance of 



the input constellation which is d^ = 2^ = 4. 



Assertion 




20 Proof of the left hand in-equality: let f^a^ and ^2 = /^^ij be two filtered 

sequences separated by the minimum distance dx, and let m be the first index for which 
a^{n)^a2{n). Since Ffz) is monic and causal, xX^)-X2{m) = ay{m)'-a^{m). Now, 

=ZkiW~^2W ^hW-^zW =KW-«2W 



25 Proof of the left hand in-equality: Let a, and ^2 be two input sequences which differ 
only in one symbol: a^{fi)-a2{n)-d^8{n), where 6{ri) is Kronecker's delta sequence, 
and denote by x. («) = / * a - {n) the corresponding filter outputs. Then, 
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sequence, -^^j-^f |=^|^f j^j/l'^)!^ • 



n 



Conclusion 



5 



The above assertion shows that monic hnear fihering improves the minimum 



distance of an uncoded PAM sequence, but not enough to justify the power increase due 
to the fihering operation. Well known pre-equalization techniques however, provide 
tools for linear filtering without suffering from a high power penalty. These tools include 
Tomlinson Harashima precoding and Trellis precoding. These tools can be employed to 
10 significantly increase dmin without increasing the transmitted power, which is exactly 
what is desired fi'om a good channel code. 

Figure la illustrates a simple ISI coder 10. The input to the coder is a sequence 
of i.i.d. M-PAM data symbols, a(n). The input symbols are assumed to be uniformly 
15 distributed across all the M constellation points. The ISI coder 10 first maps the input 
symbols sequence into a sequence of extended constellation symbols b(n) that may be 
regarded as the "state variables" of the encoder 10: 



(1.1) 



20 



where fxj denotes the integer nearest to x. The output of the ISI coder, x(n), is: 




(1.2) 



25 where the ISI generating pattern, F{z) = 1 + ^ /(/)^ 

/=i...i. 



is a monic minimum phase filter 



(i.e. has a stable causal inverse). 
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Since b(n) is drawn from the grid of odd integers, that is the same grid of a(n), it 
has the same minimum Euchdean distance, dl -d] = 4 . According to the above 

assertion, d] >dl, so ISI coding increases the minimum distance of the signal. The 

question is what is the power increase associated with ISI coding. In order to assess this 
5 power increase, we observe that (1.1)-(1.2) fold into (see Figure lb): 



x(M) = a(n)-]^g(/)^(« -/)-2M 



/>o 



a(n)-Y,gif)An-l)l{2M) 



i>0 



(2) 



where ' =F ^ (z) - 1 , which is exactly a Tomlinson Harashima (TH) precoder. 

/>0 

10 Accurate identity between equation (2) and equation (1) requires g(n) to have an infinite 
number of taps, but practically, since F(z) is minimimi phase, equation (1) can be 
approximated with equation (2) as well as desired using enough taps. It is well known 
that besides the case where the g(n) are very small, and beside singular values of g(n), 
the output of a TH precoder is a spectrally white sequence of random variables which are 

15 distributed uniformly in the range [-M,M), and the power of x(n) is higher by a factor of 

than the power of the uncoded symbols a(n). This power addition is a 

relatively small penalty for large M For example, for M= 2, 4, 8 and 16, the power 
increase is 1.25 dB, 0.28 dB, 0.07 dB and 0.03 dB, respectively. 



20 The present inventor has thus shown that the ISI coder may be a good code, since 

it is capable of increasing the distances between the transmitted sequences almost without 
any increase of the transmitted power. 



The ISI coder 10 shown in Figure la can be immediately generaUzed to the case 
25 of "square constellation" QAM modulation. In this case, a(n) are drawn from the A/^- 
QAM constellation where their real parts and their imaginary parts are statistically 
independent A/-PAM symbols. The ISI coder 10 operates according to eq. (1.1)-(1.2), 
where in this case fx] denotes the complex integer nearest to x (complex integer is a 
complex number with integer real and imaginary parts). In this case, the ISI coder's 
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output is distributed uniformly in [-M,M)x[-M,M), and again its power is higher by a 
factor of M^I[m^ - 1) than that of the ISI coder's input. 

Example - a two taps ISI filter: F(z) = 1 + /(l)z"^ , < 1 . 

5 In this case it can be shown that the minimum distance at the ISI coder 10 output 

is <i ^„ = 4(1 + |/(l)|^ ). We therefore can increase dmin by up to three dB using a two taps 
ISI filter. This example will be further analyzed in the sequel discussed herein below. 

ISI coding can be seamlessly combined with pre-equalization, by designing the 
10 ISI filter so that its convolution with the channel impulse response will be a desired ISI 

pattern. Let the channel impulse response, H{z) = I + ^hil)z~^ , be a minimum phase 

monic filter. As will be shown herein below, with an MLSD receiver, the discrete time 
channel is indeed minimum phase, since the front-end of the receiver (whitened matched 
filter) transfers all the maximum phase zeros of the channel inside the unit circle. In this 
15 case the ISI filter is F(z)/H(z), where F(z) is the desired ISI generating pattem. An 
efficient implementation of the ISI coder is shown in Figure 2 where 



b{n) = a{n)-2M{{a{n)^ fi^bin-l)- ^h{l)x{n^l))/{2M)], md (3.1) 



20 x(n) = bin) + f{l)b{n - /) - Y.^{l)x{n - 1) (3.2) 



The ISI coder for equations (1.1)-(1.2) is limited to square-QAM constellations. Square 
constellations may be non efficient when the number of data bits per symbol is not an 
even integer, and they cannot utilize shaping gains that can be achieved by non-square 
25 constellations (theoretically up to 1.53 dB and practically nearly 1 dB). In accordance 
with known techniques that generalize TH to allow shaping and a fractional number of 
bits per symbol, the present inventor generalized the ISI coder of that depicted in Figures 
la- lb to the Generalized ISI Coder (GIC) 100 shown in Figure 3. 
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The GIC 100 can be seen to consist of three functional blocks that include 
1) a linear filter 102 (either FIR or IIR), which is designed so that its convolution with the 
channel's impulse response yield a desired ISI generating pattern, F(z)\ 2) a "one to 
many" mapper S(a) 104 which maps every input symbols' sequence a into a set S(a) of 
5 multiple sequences of extended symbols, where a. the extended symbols are drawn from 
the same grid of the input symbols (e.g. ...-1,1,3,...) and thus have the same minimum 
distance, and b. the transform S(a) is invertible, that is no two input sequences are 
mapped into the same sequence of extended symbols; and 3) a Sequence selection 
algorithm 106, which selects the best sequence from the set S(a) according to a given 
1 0 selection criterion. 

Equations (1.1)-(1.2), for example, correspond to the case where S(a) is the set of 
all sequences a^lMm, where hi is a sequence of integers, and the selection algorithm 
seems to be selecting the m which minimizes the peak of the precoder output sequence. 
15 The present inventor believes that with a similar S(a)^ and a selection criterion that is 
minimization of the power at the output of the ISI coder, one may get significant shaping 
gains (A shaping gain of 1 dB was achieved using known relatively simple selection 
algorithms, that may be adopted within GIC 100). 

20 In summary, ISI coding can achieve significant improvement in the minimum 

distance of the signal, without any noticeable power penalty. The ISI coder can also 
perform pre-equalization and be cascaded with shaping algorithms such as trellis shaping. 

An intuitive explanation for the operation of the ISI coder 10 can be seen in 
25 Figure 4 by examining the dimensional vectors a, h and x of inputs, states and outputs 
of the ISI coder 10. Neglecting initial condition effects, the convolution may be 
expressed with F(z) as jc = Fb, where F is an NxN channel matrix. As demonstrated in 
Figure 4, multiplying by F transforms the sequences from the conventional "Cartesian 
lattice" to a "skewed lattice", and introduces memory into the sequence, which improves 
30 the minimum distance. Since F is an upper diagonal matrix with an all-ones diagonal, its 
determinant is one. This means that the linear transformation from A to jc is volume 
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preserving, which explains the x sequences can be packed inside the same 
multidimensional cube f-A/,A()^ that contains the input sequences perfomied in eq. (l.l)- 
(l,2). Even better than that, by using a GIC, all the jc sequences can be packed inside an 
A^dimensional sphere having the same volume of the coder's input (-MMf cube, and 
achieve a lower signal power. ISI codes can be further generalized by replacing the 
monic filter F(z) with a volume-preserving matrix F that has determinant of one, i.e.: 



where x-(x(l),„x(N)) is the coded vector, Ff^ is an NxN matrix, and b(a) is a vector of re- 
mapped inputs that is obtained from the uncoded inputs a using the S(a) sets of the GIC 
as explained above. Some desirable features for Fn are: 

a. Fn is an iVxTV^ block in Fn+i- this allows sequential generation of the encoded symbols 

(x(n) 's). 

b. Fn is stable (limy^^^ ^I^at < °^ has a stable inverse 

n 

(limyy^^ Xl^^^ ^ ^ ^"^^ ^ generalization of the minimum phase 

n 

condition, and its purpose is to ensures stability of the encoder (i.e. b(a) does not 
explode), and proper operation of the decoder. 

c. i^yvis upper diagonal; this is a generalization of the causality of F(z) and it may 
simplify the decoder design. 



Example: Time-varving ISI pattem 

/=!. ../.(«) 



6(«) = fl(n)-2M 



r 



1 



V 



J 



The present inventor believes that concatenating outputs of multiple filters may allow 
improved performance similarly to the improvements allowed by concatenated turbo 
codes. 
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Although the structure of ISI codes resembles the structure of convolution codes, 
the concept of ISI codes is different than that of convolution codes or most conventional 
codes. Conventional codes are based on constructing a set of codewords, which is a 
subset of the signal set that is matched one-to-one with the set of input sequences and has 
5 a good distance spectrum. ISI codes do not use this concept. When trying to interpret ISI 
coding in terms of conventional coding schemes, ISI coding can be regarded as a new 
multi-dimensional "lattice code": a lattice that has the same volume as the uncoded 
lattice, but a better distance spectrum. 

10 Receiver design 

In order to implement an optimal receiver suitable for working with ISI coding, 
we first assume that the channel is a Hnear Gaussian channel, whose output is given by: 

y{t)=Y,x{n)h{t-nT)^w{t) 

n 

15 

where h(t) is the continuous time impulse response of the chaimel, and w(t) is an AWGN 
(Additive White Gaussian Noise) signal. In case w(t) is not white, we can whiten it and 
factor the whitening filter inside h(t). We also assume that the z-transform of 
{n) = h{t)oh{- tj^^^^ does not have zeros on the unit circle of the z-plane; and we 

20 denote by H(z) the minimum phase component (which consists of the roots that are inside 
the unit circle) of this z-transform. It can be appreciated that in the case of an ideal 
channel h(t) is a Nyquist pulse, and thus r^,^ («) is a Kronecker delta fimction and H(z)=l, 

as one may expect. In order to achieve a desired ISI generating pattern F(z), the ISI filter 
is F(z)/H(z). 

25 

The log-likelihood of the received signal y given that the input sequence of a is 
(up to additive and scaling constants): 
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n 

where is achieved by filtering y^rj by h*(-tX T-spaced sampling and than filtering by 
l/H'^fz), and where b(n) is the extended symbols sequence that corresponds to a. Note 
that the b(n)'s depend implicitly on the last L coder states b(n) 's and Lm coder outputs 
x(n) 's prior to the beginning of the data block, which we assume that are known to the 
receiver. The MLSD can be obtained by maximizing eq. (4) over all values of a. Using a 
simpler approach, however: We refer to the b(nj 's as free variables and maximize eq. (4) 
over all values of the b(n) 's. 

The n likelihood term in eq. (4) can be determined by b(n) and by s(n)-(b(n'J), 

T 

.... b(n'L)) . This suggests a Viterbi Algorithm (VA) whose state is s(n). Some 
performance improvement and reduction in the number of states may be obtained by "x- 
range testing", that is by eliminating all the states which correspond to an 

x{n)^b{n)-\' ^f{l)b{n -/) that falls outside the legal range of the coder's output. 

When eq. (1) or eq. (2) are used, the legal range is [-MM) or f-MMMf-MM) for M-PAM 
and Af^-QAM signals, respectively. The present inventor believes that MLSD 
optimization over a and the proposed optimization over b may coincide, when x-range 
testing is used. Further investigation can be employed to verify this possibility. 

The amount of computations required per each trellis branch is similar to that of a 
typical TCM (Trellis Coded Modulation) decoder. The ISI decoder requires though L 

add-multiply operations to calculate ^f{l)b{n - /) ; however this is a relatively small 

burden, since L is typically quite small, and since the computation should be done only 
once per father state, and not once per branch. It can be appreciated that the 
computational effort can be reduced by choosing ISI coefficients f(l)../(L) that can be 
represented by a small number of bits. 



Patent Application of Ofir Shalvi 
Page 13 of 29 



y{n)-b{n)- > 

/=!.,.£ 



(4) 
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The number of trellis branches of the proposed algorithm is bounded below by M^^^ or 

fQj. ^.pAM and Af^-QAM signals, respectively. An upper bound can also be 
developed. The complexity of the algorithm may therefore be very large compared to the 
complexity of conventional decoders that achieve similar coding gains. An exception 
maybe the case of low constellations such as QPSK, BPSK or 4PAM, where the number 
of states of the algorithm may be tolerable. The algorithm in eq. (1) may significantly 
expand the constellation oib(n) compared to that oia(n)\ and that may significantly 
increase the number of states of the approximated MLSD algorithms. The present 
inventor believes that in the framework of the Generalized ISI Coders (QIC), it is 
possible to trade some noise margin for a more moderate constellation expansion. 

In view of the foregoing discussion, the present inventor proposes using a RSSD 
(Reduced State Sequence Decoder) approximation to the MLSD algorithm. Those skilled 
in the art can appreciate there are quite a few RSSD algorithms in the literature. The 
present inventor considered three classes of RSSD's including: 

1 . list algorithms : such as the M-algorithm and the T-algorithm. These algorithm are 
similar to the VA, but they sort the states according to their calculated likelihood value 
and keep only the M states that have the best likelihood values (M-algorithm), or whose 
likelihood is within a distance T from the highest likelihood in the list (^-algorithm); 

2. sequential algorithms: such as the Fano algorithm and the stack algorithm. These 
algorithms simultaneously examine trellis-paths of different lengths. They automatically 
adjust their computational complexity to confront noisy segments in the signal; and 

3. The Evuboglu & Oureshi (E&O) algorithm: This algorithm is based on partition of the 
set of all MLSD trellis states into J subsets that have good intra-subset distance. It 
maintains only one state per each subset, which is the one that has the highest likelihood 
value. 
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The present inventor mainly tested the r-algorithm and the E&Q algorithm in 
order to implement the embodiments described herein, and found that the T-algorithm 
outperforms the E&Q algorithm. The inventor's results indicated that sequential 
algorithms can achieve lov^ error probabilities at an SNR close to the cutoff rate with a 
relatively low complexity. 

Error analvsis 

The present inventor approximated the error probabilities for a signal coded by 
the ISI coder of eq. (1 . 1)-(1 .2) in the presence of AWGN with spectral density Nq as 
follows (Eqs 5 and 6 below are beUeve to also be a reasonable approximation for the 
error probabilities of the other ISI coders discussed herein (i.e. the GIC and the case of 
channel's ISI)): Consider first a receiver based on the Quasi-MLSD algorithm. The 
Event Error Rate (EER) in this case has been shown to be approximated by: 



where the summation in eq. (5) is over all possible errors sequences e in the coder's states 
b(n), Q is the well-known Gaussian erfc function, 



is the distance of the error event, and P(e) is the probability that the error e generates a 
sequence that is allowed by the receiver. P(e) = 1 if the decoder does not employ x-range 
testing. In case x-range testing is employed and the ISI coder is eq. (1) or eq. (2), the 
present inventor believes that: 



^(^) = LI/^(^ " |r^a/(e(z))|/2)[/(M -|zmfl^(e(0)|/2)/M' , where U(x)=x if jc>0 




(5) 



and 0 otherwise. 
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The Symbol Error Rate (SER) of the MLSD algorithm may be approximated by: 
PssR ^ y2Y.He)P{e)Q(d{e)/{2NX) (6) 

e 

5 where N(e) is the number of erred symbols in the error sequence e. 

Typically the sums in eq. (5) and eq. (6) will be dominated by the error sequences 
with the minimum distance, dmin • However, as shown herein below, in many cases the 
rest of the error spectrum should not be neglected. 

10 

The error probability of list and sequential RSSD is typically dominated by 
Correct Path Loss (CPL) events, in which the algorithm excludes the correct path from its 
list (or stack). These error events are characterized by long error bursts. 

15 Consider now a receiver based on the E&Q RSSE algorithm. This case is more 

difficult to analyze than the MLSD decoder, since here the error rate is increased due to 
early merging events. In order to analyze the error probability of E&Q RSSE [E&Q], we 
assume that the constellation of each symbol in the VA state vector is divided into j. 
Ungerboeck cosets and that the set of all VA states is divided to J=j^~^ subsets 

20 correspondingly. The error event probability can be approximated by eq. (5), where in 
this case: 

a. The sum goes over all non-zero error sequences of the type e = (e(l) . . . e(N)), 
which starts with L zero elements (to eliminate propagation from previous error 
25 events), and whose last L elements are a product of a complex integers with the 
spacing vector of the Ungerboeck coset which is used to partition the set of VA 
states; and 



30 
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b. ^(e) in eq. (5) is replaced by 



11=1 ...A^ 



^W+Z/(^M'^-M , where 



5 (the error energy excludes the effects at the output of the coder beyond «=A9. The error 
events were divided by [E&Q] into two types including: 



1. MLSD error events - The last L elements of e are zero: these are errors that even the 
MLSD would have; and 

2 CPL error events - One or more of the last L elements of e are non-zero: In these 
errors, the RSSE throws the true state from its trellis, which will typically lead to a long 
tail of errors. 



15 As discussed herein before, and as the present inventor also observed in simulations, the 
algorithm suffers from severe error propagation, and due to this phenomenon, eq. (6) is 
typically not a good approximation in the case of RSSD. 

Example revisited ■ a two taps ISI filter : f(z) = 1 + /(l)z'^ , |/(l)| < 1 • 

20 As stated above, the error events e, = 7*(...020...), k = 0...3 achieve the 

minimum distance dmin = 4[l + This suggests that choosing |/(/)| » 1 may earn 3 
dB. However, error events with n errors of the form 

e„ = (.. .0,2/7,2/?^,.. .,2/7",0...), k = 0..3, where p is the nearest among -7, -j} to 
f(l) in the case of QAM, or simply -sign(f(l)) in the case of PAM, have distances 
25 d^{e„) = 4(1 + ^ + - l)(j/? - /(/ ). So as 1/ (/ J approaches 1 , dmin increases, but 

very many error events may become likely. For example, for f (I) =0,99, dmin is better by 
2.97 dB than that of the uncoded sequence, but there are 5000 error sequences ^2 - -^sooo 

whose distances lie within 1 dB from Jm/Vi^- This implies that /(/) « 1 may be a poor 
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choice. For QAM signals, this problem may be avoided by choosing f (I) ^Ae/"^"^ where 
A^l A<1, which may yield the expected 3 dB gain. In addition to that, x-range 
testing may help combating such long error events. 

Analysis and Optimization of some ISI Patterns 

The distance spectrum of some ISI patterns that seem to have very high coding 
gains are now discussed herein below. 

Distance Spectrum Calculation 

The present inventor first developed an algorithm to compute the distance 
spectrum of the ISI code. A brute force approach would be to calculate the distances of 
all possible error sequences and sort out the minimum distances, but this approach would 
require a non-practical amount of computations. Instead, the present inventor used an 
approach proposed by others that is guaranteed to find any error sequence e whose length 

is below Nn^ax and whose distance is below dtound + ^filf where dboun/ and A^;;,^, are 

parameters of the algorithm. The algorithm iteratively builds a tree of error sequences 
whose leafs are ej^=(e(]) ...e(N)X sequences of error terms (i.e. sequences of even 
complex valued numbers). The offspring of each leaf ^yv is all the error sequences whose 
first elements are ej^. The idea behind the algorithm is that it can eliminate all the 
offspring ofa leaf whose is high, since dj,{e^) always increases with A^ as 

we travel along a branch of the tree. The algorithm eliminates all leafs such that dl{e^) 

> dbound , and as a result the tree reduces to a finite size. The algorithm does not expand 
the offspring of error sequences whose last L elements are zero. The Euclidean distance 

of an error sequence d\eN)> dl (cn) since d^(e!^ only measures the first A^ 

errors at the coder output, and since any with non-zero last L elements creates a tail of 
errors for n>N at the coder output, and the magnitude of its last non-zero element in this 
tail is at least |2/(z)j . The algorithm outputs the distances d^(eN) of all the tree elements 

(most of them have very high distances). The dtomd parameter should be chosen high 
enough so that the minimum distance of the code and all the portion of its distance 
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spectrum that significantly effects the error probability are below dbom/ + 4|/(z)]^ . 
However, a too high di^ound may yield a huge tree size. Nmax should be chosen such that 
(^n) ^ d)jound for any error sequence of length N'=^Nniax (i.e. there are no error 

sequences of length Nmax in the tree). Trial results discussed herein below used Nmax~l20, 
5 and are described for the cases where the tree had error sequences longer than Nmax 
symbols. 

Finding good ISI Patterns 

The problem of fmding the best (or worst) ISI in terms of optimizing dmin under 

10 power constraint has been addressed in the art by others, which have found that both the 
best and worst ISI is achieved when all the zeros of the system are on the unit circle of 
the z-plane. The present inventor recognized a different problem that does not operate 
under power constraint, and believes that it can be shown that certain ISI patterns whose 
zeros lies on the unit circle are (at least local) maxima of the minimum distance 

15 optimization criterion. The present inventor has found out that ISI patterns that have a 
deep null near a frequency of 0, ±7r/2, or % have pretty good distance spectra. Table 1 

below shows the distance spectra of ISI pattems that have L zeros at z = Ae"^ . The 
second column shows the number of zeros of the ISI pattern. The third and the fourth 
columns show the amplitude and the phase of the zeros of the ISI pattem. The fifth and 

20 the sixth columins show the distances and the number of erred symbols of the minimum 
distance error events. The seventh column shows the expected SNR gain at an event 
error rate of 10"^ compared to an uncoded 64-QAM signal, calculated using eq. (5) 
assuming P(e)=l, and taking into account the 0.07 dB power increase due to ISI coding. 
The bold letters show the gain taking into account only the dmin event, while the gain in 

25 regular letters takes into account all the distance spectrum that was mapped (in some 
cases only a small portion of the spectrum was mapped). The eighth column shows the 
lowest distance that may be missed by the analysis algorithm. The ninth column is the 
size of the tree spanned by the algorithm, and in parenthesis we see the number of 
elements of length Nmax symbols that were left in the tree. 

30 
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Looking at patterns with L=2 zeros, we observe that even such short ISI codes can 
achieve hefty gains of 5-6 dB. Note the set of error sequences -^/C- 0, 2, -2, ...,2('l)^ 
,0 .,.), k=0...3, /w=7,2,..., whose distances decay pretty slowly with m (e.g. in the case of 
5 ISI pattern 2, d (em) = 15,01 + 0.0865m). These error sequences degrade the SNR gain of 
the code by 0.5-0.8 dB. 



Looking at patterns with L=3 zeros, we observe the gain of ISI codes can surpass 
7 dB. If we add to that a shaping gain of 1.53 dB, assuming an optimal shaping 

10 algorithm, these results imply that we can improve the SNR of QAM by nearly 9 dB. 
Now, it is well known that the gap between the SNR budget required by QAM to achieve 
an error rate of 10"^ and the SNR required according to Shannon bound to communicate 
reliably at the corresponding data rate is also 9 dB. This means that a receiver that 
approximates the MLSD performance for such ISI patterns can operate at SNR levels that 

15 are within fractions of a dB of the Shannon capacity. However, there is a disclaimer to 
this ambitious statement. As the gain of the ISI pattern exceeds 7 dB, the tree of the 
Aulin algorithm starts to grow to mammoth sizes. This may imply that it would be very 
difficult to approximate an MLSD receiver with a practical algorithm. In any case, as 
shown in simulations, RSSD algorithms are capable of achieving a 5.5-6 dB gain and 

20 approaching the cutoff rate when the high gain ISI patterns of table 2 shown below are 
used. 



Looking at ISI patterns with L-4 zeros, we observe that a mild spectral null, 
i4=0.7-0,8, gives a gain of 5.5-6.5 dB. For the deeper nulls, >4=0.85-0.86, the error tree is 
25 so large that the present inventor could not analyze the distance spectrum beyond finding 
dmin- For the deepest nulls, A>0.S6, the error tree was so large that the present inventor 

could not even find dmin- However, dmin^ can be lower-bounded by d^oun/ + 4|/(z)|^ 

which corresponds to impressive gains of 7.5 dB. Distances of error events that were 
found by algorithm bound dmin from above. 
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The present inventor observed a neat symmetric structure of the low distance error 
sequences: In an error sequence of N errors, e(N-n) is equal e(n) up to a possible 
conjugate operation and multiplication by /, -l.j or -/ A similar error sequence structure 
(but for the real valued case) has been shown by others skilled in the art. 

Analysis and Optimization of the E&O RSSD 

The ISI patterns in table 1 have good distance spectra for an MLSD decoder, but 
when the E&Q RSSE decoder is used, they may suffer from CPL error events that have 
significantly lower distances. The E«&Q RSSD algorithm is generally outperformed by 
the list and the sequential RSSD algorithms, but the prior analysis of its error probability 
by others allows optimizing the ISI pattern towards improved RSSD performance. 

Although the Aulin algorithm has been shown by others that it can be modified to 
analyze the error events of RSSD, the present inventor took a simpler approach: The 
RSSD algorithm was run at low SNR conditions and the errors events were mapped at the 
decoder's output. Table 2 below summarizes the distances of RSSE error events for an 
ISI pattern which has two zeros at 0.9^^'^. The last columns in Table 2 show the 
minimum distance error sequence that leads to a CPL event. 



Table 2 
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As shown in Table 1, for this ISI pattern, a minimum distance of 14.81 and a gain 
of 4.9 dB would be obtained by an MLSD decoder. However, as can be seen from Table 
2, the gains of an E&Q RSSD detector with this ISI pattern are only 3,4, and 4.4 dB for 
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and 16 cosets, respectively. Moreover, since the RSSE errors are dominated by 
CPL error events, the E&Q RSSD decoder is characterized by very long error bursts for 
this ISI pattern. In order to address this problem, the present inventor developed an 
iterative algorithm to search ISI patterns that enhance the performance of an E&Q RSSD 
5 decoder. This algorithm iterates between mapping the distance spectrum of the ISI 
pattern and steepest descent adaptation of the ISI pattern taps to minimize the Symbol 
Error Rate determined via eq. (6). These events are given high weights N(e) in order to 
avoid CPL events. Although this algorithm is still under construction, it has been found 
by the present inventor to reach non-stable ISI patterns that have zeros outside the unit 

10 circle. Thus, even after this problem has been resolved, there is no guarantee that the 
SER criterion of eq. (6) does not have local minima that will cause the algorithm to 
converge to sub-optimal solutions. Table 3 shows the distance spectra of some ISI 
patterns that have been optimized for RSSD decoding. The table shows the ISI pattern, 
the parameters of the RSSE decoder, and the distance spectrum for both MLSD errors 

15 and RSSE exclusion errors. Again, the table shows the SNR gain at an event error rate of 
10'^ compared to an uncoded 64-QAM. The average number of erred symbols per error 
events is also shown (assessed via simulations). The first ISI pattern allows a gain of 1 .7 
dB, with a low complexity RSSE decoder that has only 4 branches in its trellis. Note that 
this decoder detects a P AM symbol (or a real/imaginary component of a QAM symbol) 

20 per iteration, while all the other decoders detect a QAM symbol per iteration. The second 
ISI pattern was selected by a grid search among the entire stable single zero ISI patterns. 
It achieves a gain of 2.5 dB with an RSSE decoder that has a trellis of 16 branches. Note 

that by increasing |/(/)| towards 1, the minimum distance of the algorithm is increased 

from 7.61 towards 8; however on the other hand, the distances of the RSSD CPL errors 
25 are reduced. The third ISI pattern was achieved by some optimization over two zero ISI 
patterns, and it achieves a gain of 3.5 dB with an RSSD trellis of 64 branches. It can be 
seen that these ISI codes and E&Q RSSD are still less desirable than at least one other 
known TCM in terms of their performance versus complexity tradeoff, but not by far. As 
shown in the simulations, the section T-algorithm RSSD and sequential RSSD 
30 outperform the E&Q RSSD as well as this at least one other known TCM. 
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Tables 
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Simulation results 

The present inventor tested the ISI coder in eq. (1) for the case where the channel 
is ideal and the receiver is the E&Q RSSE algorithm without any x-range testing. The 
results are discussed herein below with respect to Figures 5-7. 

Simulation test 1 - Small Code Gain. Simple E&Q Decoder 

Figure 5 shows the Event Error Rate (EER) versus the SNR (Es/NO [dB]) 
according to eq. (5) for an ISI pattern F(z)=l+0.8z'' and an 8-PAM input. The figure 
shows the EER obtamed by an E&Q RSSD decoder with J=2 cosets. It can be seen there 
is an excellent match between the analytic curve (bold line) and simulation results (in '*'). 
hi this case, even such a simple decoder that has only 2 states and 4 branches, is capable 
of gaining the full coding gain of the ISI code, which is about 1.75 dB at error rate of le" 
. According to eq. (5), employing x-range testing would only reduce the EER by 30- 
40%, which is equivalent to only a 0.1 - 0.15 dB improvement in the coding gain. 

Simulation test 2 - Larger Code Gain. More CompUcated E&Q Decoder 

Figure 6 shows the Event Error Rate (EER) versus SNR according to eq. (5) for 
an ISI pattern F(z) that has two zeros at z=-0.9^^^, and a 64-QAM input. The figure 
shows the EER obtained by an E&Q RSSE algorithms with J=4. J=8. andJ=J6 cosets. 
It can be observed that the moderate complexity algorithm with J=4 achieves a coding 
gain of about 3 dB, and the more complex RSSE algorithms with ^8,16 achieve coding 
gains of about 4 dB. This is in agreement with the analytic error predicted in table 2 (up 
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to 0.2 dB). Since the RSSE errors are dominated by exclusion of the correct path from 
the trellis, the algorithm suffers from long error bursts. The average number of erred 
symbols per error burst was in the range of 30-70 for 7=8, 16, and 500(!!) for 7=4, 

5 Simulation test 3 - J-algorithm Decoder 

Figure 7 shows the error event rate versus Es/NO of a T-algorithm RSSD for an 
ISI pattern that has three zeros at z—O.QSe!'^^, According to Table 2, this ISI pattern 
yields a gain of 6.7 dB. The numbers in "()" are the average number of paths per symbols 
and the list size. 

10 

The vertical lines show the Shannon bound for shaped (Gaussian) signals at a 
SNR of 18 dB; and 1.53 dB above it is seen the Shannon bound for an un-shaped signal; 
and another 1.68 dB higher is seen the cutoff rate. 

15 It can be seen that reasonable complexity provides an error probability of le'^ at 

SNR levels of 21 .5 dB. This is 0.3 dB above the cut-off rate and 5.7 dB better than an 
un-coded signal. For a SNR of 21 dB, which is 0.2 dB below the cut-off rate, there is a 
dramatic increase in the average and the maximum list size; and yet the algorithm was 
not able to reach an error probability below le'^. 

20 

Simulation test 4 - Sequential /"-algorithm Decoder 

Figure 8 shows the error event rate versus Es/NO of a sequential T-algorithm 
RSSD for the same ISI pattern used in Figure 7. The sequential T-algorithm is a T- 
algorithm that automatically senses an error event (by a sudden rise in the average list 
25 size), and than goes back in time and re-process the potentially erred signal segment with 
a higher Tand a higher list size (which allows reducing the CPL probability). Again, the 
numbers in "Q" th^ average number of paths per symbol and the maximum list size 
(in case of increased T). This yields a very efficient algorithm in term of its average 
number of operations, as long as the SNR is higher than the cutoff rate. 

30 
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In view of the above, it can be seen the present invention presents a significant 
advancement in the channel coding art. It should be apparent that the present invention 
represents a significant departure fi:om the prior art in construction and operation. 
However, while particular embodiments of the present invention have been described 
5 herein in detail, it is to be understood that various alterations, modifications and 

substitutions can be made therein without departing in any way fi'om the spirit and scope 
of the present invention, as defined in the claims which follow. 
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